The invention relates to a method for augmented depiction of at least one piece of additional information in at least one image of a surrounding environment, in particular a vehicle surrounding environment. Furthermore, the invention relates to a device for the implementation of the method.
Devices for augmented depiction superimpose a generated piece of additional information with a recorded and perceptible surrounding environment. The pieces of additional information are thereby depicted in the correct position in the images of the surrounding environment, wherein the images of the surrounding environment are output in the background. Such devices are, for example, partially transparent display units such as a HUP display unit (HUP=Head Up Display), a HMD display unit (HMD=Head Mounted Display), or opaque display units, for example a conventional screen.
Such a device for augmented depiction is, for example, known from DE 10 2011 115 739 A1.
The object of the invention is to specify an improved method and an improved device for augmented depiction of at least one piece of additional information.
In the method according to the invention for augmented depiction of at least one piece of additional information in at least one image of a surrounding environment, in particular a vehicle surrounding environment of a vehicle, the image forms a background of a display, wherein several pieces of additional information to be displayed are combined into at least one group and are output to be superimposed in the background as a piece of group information.
The invention enables a continuous adaptation of the overlaying of virtual pieces of additional information. In particular, a quickly recognizable allocation of virtual pieces of additional information to real objects in the real surrounding environment is enabled, in particular in dynamic scenes, such as video sequences, which are recorded during a drive with the personal vehicle and are output on a display. Furthermore, the vehicle enables intuitive interactions with the grouped virtual pieces of additional information for recognition of the content of all or only relevant or selected pieces of additional information by an observer.
The invention is applicable in the vehicle for detection and depiction of a vehicle surrounding environment. Additionally, the invention is also applicable for mobile devices, such as a mobile telephone, smart glasses and/or for virtual displays such as a projection screen and for further purposes, such as advertising, pedestrian navigation or for displays in means of transport, such as a bus, train, plane and ship. An adaptation of the superimposed pieces of additional information occurs here depending on the background and/or depending on an event and/or situation.
In a development, several pieces of additional information to be depicted are output as a piece of group information in such a. way that the individual pieces of additional information are output one over the other and partially or completely cover one another. By manual or event-controlled selection of an uncovered region of a covered piece of additional information, this can be output in the foreground as a first piece of additional information. For example, previously covered pieces of additional information can be output in the foreground upon reaching the associated real object in an event-controlled manner and/or depending on distance, in particular pieces of additional information from nearby real objects, and/or depending on a predetermined relevance and/or a priority. Therefore, a ranking or priority sequence is possible in the depiction of several pieces of additional information, wherein the ranking or priority sequence is able to be predetermined manually or in an event controlled manner and/or depending on the situation.
Alternatively or additionally, pieces of additional information to be depicted in the future can be compiled into a first group and output as a first piece of group information. Therefore, for example, objects lying at a great distance from the reference point, in particular from the current position of the personal vehicle, are compiled into a first group. With the approach of the personal vehicle, these grouped pieces of additional information are then able to be read completely by the personal vehicle, in particular on reaching the real positions thereof, and for example are output separately in the background one next to the other and/or one over the other.
Furthermore, it is provided that alternatively or additionally, pieces of additional information to be depicted currently are compiled into a second group and output as a second piece of group information. This gives the observer the opportunity, for better clarity of the display, for example, to only output those virtual pieces of information which are lying ahead in time and/or location and therefore lie in the future. The virtual pieces of additional information to be depicted currently are compiled and output, for the reduction of the density of information in the display, to lie partially or completely one over the other in the piece of group information.
Furthermore, it is possible that previous virtual pieces of information, for example depicted in the past, are compiled into a third group and are output as a third piece of group information. The observer can hereby, when needed, obtain a review of the virtual pieces of information depicted in the past by a simple selection of the third piece of group information.
In one development, the pieces of additional information of at least one of the groups are compiled in a sorted manner according to the distances thereof from the position of an observer of the display, in particular from the position of the personal vehicle, and/or according to the associated display duration in the background on the display and are output as a relevant piece of group information. Therefore, the pieces of additional information belonging to a group can be compiled in a sorted manner according to the distance from the reference point, for example the personal vehicle, and if necessary output such that the output of the pieces of additional information occurs in a sequentially correct manner.
The invention additionally enables a dynamic interaction with the display, by the piece of additional information being output in the background being able to be manipulated manually by means of at least one selection function and the output thereof in the background being able to be changed or adapted accordingly. In particular, the output and/or the manipulation of the piece of additional information to be output can occur by means of a voice-, image- and/or button-controlled function.
In one possible embodiment, the piece of additional information is output as an additional object and/or an additional text, wherein at least one of the additional objects is generated as a virtual road sign with or without reference (contextual reference) to the depicted surrounding environment. In particular during application of the invention in a display of a vehicle, the current position of the personal vehicle serves as a reference point.
Furthermore, it can be provided that the additional piece of information is output in particular in an event-controlled manner and if necessary alternately as a 2D depiction and/or as a 3D depiction. Therefore, for example, a piece of additional information output as an additional object, for example a virtual road sign, can, for example, be overlaid in the background in a 3D depiction and/or a piece of additional information output as an additional text, in particular a labelling of a real or virtual object in the output image, can be overlaid in the background in a 2D depiction. This increases the clarity of the display and enables an improved perception and ability to recognize relevant information in the background.
With regard to the device for the implementation of a method, this comprises at least one recording unit to record and detect an image of a surrounding environment which forms a background of a display; at least one processing unit for graphic processing of a recorded digital image and transformation of the digital image into an output image as well as to generate and superimpose at least one piece of additional information into the background; and at least one output unit to output the piece of additional information into the background.
The recording unit is, in particular, one or more cameras, in particular a stereo camera system, an omnidirectional camera, an infra-red camera, a laser-scan unit, a radar, a lidar or a combination of comparable detection units. The processing unit can be integrated. into the recording unit or can be formed separately. The processing unit can be an integrated switch or a control device or another suitable image processing unit, The output unit can be a classic screen, a HUP display unit (HUP=Head Up Display), a MMD display unit (HMD=Head Mounted Display) or another suitable display unit.
In a further method according to the invention for augmented depiction of at least one piece of additional information in at least one recorded digital image of a surrounding environment of a vehicle:                the recorded digital images;        the associated detection time and        at least as a vehicle parameter, an associated current position of the vehicle        are stored in an event-controlled manner, wherein the storage of the digital images occurs in a limited manner in terms of location and/or time, depending on the detected event and the recorded digital images, and/or wherein the stored digital images of at least one event are output on a display as an output image.        
The advantages achieved with the invention in particular consist in that an intuitive perception of the relationship between the real surrounding environment and the virtual pieces of additional information, and in particular better clarity due to the compilation of several virtual pieces of additional information to be depicted into pieces of group information, are enabled. Additionally, a sequentially correct and therefore chronological depiction of virtual pieces of additional information is enabled in the display. Furthermore, the display can be adapted according to an event and/or situation and pieces of additional information regarding a certain event can be output separately.
During application of the method for augmented depiction in a vehicle, the virtual pieces of additional information correspond directly to the surroundings of the vehicle surrounding environment which are visible to the driver, wherein the “reduced” augmented depiction of the pieces of additional information according to the invention, such as additional objects/additional text, occurs by compilation of several pieces of additional information into sorted groups. Pieces of additional information that are to be depicted/are depicted can thereby be compiled into the following groups:                preview group (=first group): pieces of additional information, the display of which is imminent;        current group (=second group): pieces of additional information which are currently visible in the vehicle surrounding environment;        review group (=third group): pieces of additional information which were recently visible in the display.        
The following sorting is possible within a group:                sorting of the pieces of additional information according to distance from the reference point, the vehicle;        sorting according to the dwell time of the pieces of additional information in the display.        
For temporal and content interaction and manipulation of the output/imaging of the pieces of additional information in the output image of the surrounding environment, various functions, such as “stopping a currently output sequence of pieces of additional information and/or output images” (=stopping of the relevant video sequence), “rewinding”, “piece of group/additional information” and “interior view” are offered to the passenger.
Exemplary embodiments of the invention are explained below in more detail by means of drawings.